Skip to content

Add Kimi-K2.5 support#19170

Open
AesSedai wants to merge 6 commits intoggml-org:masterfrom
AesSedai:kimi-k2.5
Open

Add Kimi-K2.5 support#19170
AesSedai wants to merge 6 commits intoggml-org:masterfrom
AesSedai:kimi-k2.5

Conversation

@AesSedai
Copy link

@AesSedai AesSedai commented Jan 29, 2026

Adding support for https://huggingface.co/moonshotai/Kimi-K2.5

Since this model includes compressed-tensors (INT4 for the conditional experts), I moved the dequant_model to the prepare_tensors call at @compilade's suggestion. The model conversion fails otherwise because the quantization_config is nested under the text_config in the config.json.

Additionally, this model adds some new keys for the vision tower, prefixed as vt_, and the preprocessor_config.json has the expected fields nested in the media_proc_cfg key.

This PR does not include the "hacked" Q4_0 changes by @jukofyork, referred to in this comment.

I have added a first pass at vision support, heavily aided by LLM assistance. I entirely expect @ngxson to tear it to shreds or call me a dummy and show me an easier way to add that vision support :)

@AesSedai AesSedai requested a review from CISC as a code owner January 29, 2026 06:28
@github-actions github-actions bot added the python python script changes label Jan 29, 2026
@AesSedai
Copy link
Author

AesSedai commented Jan 29, 2026

image

I was testing the VL locally with this image, and the thinking block shows the oddness:

This is an interesting image. The user is asking me to identify what's in the picture. Let me look carefully.

The image shows what appears to be a furry animal resting on a tree branch. It looks like it has grayish-brown fur, and there are distinctive white markings on its face/ears that look like little tufts or tassels. The animal appears to be sleeping or resting with its face tucked in.

Based on the appearance - specifically the white ear tufts and the overall body shape - this looks like a **Common Marmoset** (Callithrix jacchus), also known as the White-tufted-ear Marmoset. These are small New World monkeys native to Brazil. The white ear tufts are their most distinctive feature.

However, I should also consider if it could be something else. Could it be a raccoon? No, the body shape is wrong and raccoons don't have white ear tufts like that. Could it be a type of lemur? Some lemurs have ear tufts, but the body shape here looks more like a marmoset or tamarin. The Common Marmoset fits best with those prominent white ear tufts.

The animal is curled up on a branch, likely sleeping or resting, with its tail visible (appears to be a long, furry tail). The background is green foliage, suggesting a natural or semi-natural habitat.

So my answer should identify it as a Common Marmoset or White-tufted Marmoset, pointing out the distinctive ear tufts and the fact that it appears to be resting on a branch.

Wait, looking closer at the image... is it possible this is multiple animals? Let me check. The pattern seems to show overlapping forms. Actually, looking carefully, this might be TWO marmosets huddled together? Or is it just one curled up?

Actually, looking at the fur pattern and the way the bodies are arranged, this appears to be two common marmosets huddled together on the branch, sleeping. I can see what looks like two heads with white ear tufts close together. Yes, that makes sense - they often sleep huddled together for warmth and security.

So the answer is: Common Marmosets (White-tufted Marmosets), likely two of them, sleeping or resting on a branch.

I should mention:
- Species: Common Marmoset (Callithrix jacchus) or White-tufted-ear Marmoset
- Distinctive white ear tufts
- They appear to be sleeping/resting on a tree branch
- Possibly two individuals huddled together
- Native to Brazil
- Small New World monkeys

This seems accurate based on the visual evidence.

Specifically, this line lead me to believe there's a problem:

The pattern seems to show overlapping forms

and I've seen that in other local generations with this image too, eg:

Actually, looking at the pattern again, this looks like it might be multiple images stacked or a glitch effect? No, looking closely, it appears to be a single image of a raccoon resting on a branch, but there might be some artifacting or the image is showing the raccoon in a specific pose. The repetition pattern suggests it might be a glitch or the image is tiled strangely? No, actually, looking carefully, it seems like the image might be corrupted or displaying incorrectly, or it's an artistic effect?

Wait, no, looking at the image again - it appears to be a raccoon lying on a tree branch, and the image quality or compression creates a somewhat repetitive pattern, but it's clearly a raccoon. The distinctive facial markings (black mask, white nose/forehead), the gray fur, the bushy tail possibly visible - this is definitely a Common Raccoon (Procyon lotor).

In comparison, this is a bit of the thinking from the OpenRouter API for Kimi-K2.5:

The user wants to know what's in the picture. Looking at the image, it's clearly a raccoon lying on a tree branch. The raccoon has the distinctive black mask around its eyes, gray fur, and is draped over the branch in a relaxed or tired pose. The background shows a forest or wooded area with green foliage.

This is a straightforward image description task. I should identify the animal correctly as a raccoon and describe what it's doing (resting on a branch). I don't need to overcomplicate this or add fictional elements since the user asked a direct question about the image content.

Vastly different feel in the confidence of its answer based on what the VL sees.

@CISC
Copy link
Collaborator

CISC commented Jan 29, 2026

While the mmproj conversion appears to work and the model loads and can decode images, I've got some weird output when using the vision component that leads me to believe there is a conversion issue somewhere or some other missing component. I think I need some review from @ngxson to help get it working correctly.

Yep, seems something is not quite right yet.

**cfg["media_proc_cfg"],
}
# merge configs
self.preprocessor_config = {**self.preprocessor_config, **cfg}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

self.preprocessor_config is empty at this point, so not really necessary to merge, but will allow it for consistent looks.

Add new kimi-k2.5 keys to mtmd convert
Update V_MMPROJ tensor mapping for new mm_projector.proj keys
Update V_M_IMP_NORM for new mm_projector.pre_norm key
@AesSedai
Copy link
Author

AesSedai commented Feb 1, 2026

Vision is working now for images, uploaded MMPROJ files to my repo.

@ngxson I left comments about the places that confused me the most.

  1. the resize_position_embeddings_3d - might be combinable with the clip_graph::resize_position_embeddings if the tensors are handled better?
  2. clip_graph::build_rope_2d_interleaved roughly makes sense to me from a 10,000 foot view, but I was thinking that maybe zipping or transposing the pos_w / pos_h tensors might make the square peg fit in the round hole with a bit of a different math approach?
  3. I have no idea why inp = ggml_add(ctx0, inp, learned_pos_embd); wasn't working in build_vit by passing in the learned_pos_embed.

I think the rest of the changes are pretty sane.

@AesSedai
Copy link
Author

AesSedai commented Feb 1, 2026

Some test samples that I ran locally. A very basic OCR test:
chatlog (13)

A more complicated OCR test that includes transcription:
chatlog (14)

And the interpretation of the raccoon photo from earlier:
chatlog (15)

The two things that concern me still are:

  • Image 1: there is a mention of a "vertical black line/border on the left side" in the thinking, plus mention of a "Border: There is a thick vertical black line running along the left side of the image" in the response. The image padding is black, so perhaps something related to that?
  • Image 3: In the thinking, item 6 mentions: "There's a visible seam or line in the image, suggesting it might be a composite or stitched image, or perhaps just an artifact". There isn't a seam like that, so I'm concerned.

@AesSedai AesSedai marked this pull request as ready for review February 1, 2026 11:27
@AesSedai AesSedai requested a review from ngxson as a code owner February 1, 2026 11:27
@segmond
Copy link

segmond commented Feb 1, 2026

Great work AesSedai! I just downloaded the BF16 for mmproj. Is there any reason to get anything higher than Q8_0? What about ctk/ctv is there any good reason to run them in f16 instead of lower since the model is INT4?

@segmond
Copy link

segmond commented Feb 1, 2026

I'm happy to report that I have tested this branch and it works great. I ran it with the Q4_X quant and my ctk/ctv at q8_0. Using the BF16 mmproj.

Screen Shot 2026-02-01 at 12 31 36 PM

@AesSedai
Copy link
Author

AesSedai commented Feb 1, 2026

@segmond Thanks, for the MMPROJ some cards are more or less compatible with different versions. The BF16's don't work very well on my 3090s IIRC. The Q8_0 should be fine to use quality-wise.

Regarding CTK / CTV, you do not want to quantize the cache on this model at all. The model weight quantization is different than the cache quantization. MLA / GQA already comes with some pretty severe compression on the cache so by further quantizing it you'll degrade it more. Besides, the context is very lightweight anyways. Something like 165k context in FP16 is like ballpark 10GB or so.

@tempgidam
Copy link

Hello,

thank you for your work.
I've been testing this PR on my own server and I'm still getting some odd comments/observations from the model running some vision tests.

In my most basic test (showing the model an image of a simple black circle on a white background), my local version always insists that it's seeing two circles. When I counter-check the same image using K2.5 over openrouter (provider limited to MoonshotAI for consistency), this straight up never happens.
It does this for pretty much any image showing a simple circle that I've tried..

With more complex images like photos of people that I know the model recognizes or random artwork, my local variant consistently sees images as stitched, fragmented or artifacted with seams or similar things going through them. For any given image, the artifacts/effects my local version seems to see are consistent so it'll always note the same things over several attempts.
Again, K2.5 via the API doesn't show any of these problems for me.

In general, asking the model to comment on the image quality seems to reliably reproduce this issue. When I first tried the raccoon image above, the answer actually seemed fine at first glance. But adding an inquiry about quality makes it clear that the model is seeing some heavy issues like lines and even claiming that it appears to be rotated.

local w/o prompt about image quality

image

local w/prompt about image quality

image

api w/ prompt about the image quality

image ____

Here's two more examples, but as mentioned, the issues are present across numerous retries and several different images/styles of images.

local

image

api

image ____

local

image

api

image

I'm running your Q4_X quant and the bf16 mmproj. I built this PR from scratch just a few hours ago so it's the most recent version. I'm not using any form of kv-quantization.

Here's what I'm using to load the model:
./llama-server-k2.5 --model Kimi-K2.5-Q4_X-00001-of-00014.gguf --ctx-size 64000 -ngl 99 --n-cpu-moe 99 -ub 4096 -b 4096 --threads 32 --parallel 1 --host 0.0.0.0 --port 5001 --jinja --chat-template-file chat_template.jinja --special -mm mmproj-Kimi-K2.5-BF16.gguf

For testing purposes, I'm not running any sort of system prompt or similar in Sillytavern so that the only thing the model processes is my question and the image.

Sorry for the image spam but I hope this helps.

@AesSedai
Copy link
Author

AesSedai commented Feb 3, 2026

@tempgidam Huh, interesting. I haven't seen any double-vision issues on my setup. I did update the first shard yesterday evening to adjust the chat template since MoonshotAI updated the media token. There's a minimal prompt on my ST setup ("A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions.").

There was also a fix for the thinking block issue in the chat template that required an update a couple of days ago. Could you download a fresh copy of the first shard of the quant and re-test?

I'd be surprised if there wasn't a bug lurking in there somewhere, which is why I'm waiting on ngxson to review.

@z80maniac
Copy link
Contributor

I used AesSedai's model (IQ2_XXS + Q8_0 mmproj) and tempgidam's image of a circle. Then prompted this:

Look at this picture and create an approximate SVG file of what you see.

And it generated me this:

<svg viewBox="0 0 400 300" xmlns="http://www.w3.org/2000/svg">
  <circle cx="100" cy="150" r="80" fill="none" stroke="black" stroke-width="15"/>
  <circle cx="320" cy="150" r="80" fill="none" stroke="black" stroke-width="15"/>
</svg>
This is how it looks in Inkscape image

I had to disable the thinking, because with thinking enabled it just rambled on and on for 3000 tokens, endlessly correcting itself, and then gave me an unusable SVG in the end. However, in its ramblings it said this:

The left circle appears to have its left side cut off by the image boundary

...which doesn't correspond to the image above, but still mentions that one of the circles is cut off. Only in the image, the right circle is cut off slightly. So maybe there is something to it, or maybe it's just a hallucination because the quant is too small.

The whole thinking process of the failed run (which was NOT used to generated the above image)

The user wants me to create an approximate SVG file of the image shown. The image displays two black outlined circles (or ring shapes) on a white background. They appear to be positioned side by side, with the left one partially cut off on the left side and the right one fully visible or partially cut off on the right. Actually, looking more carefully, it seems like two circles with thick black strokes, positioned horizontally next to each other.

Let me analyze the image:

  1. Two circles with thick black borders
  2. The left circle appears to be cut off on the left side (or it's just positioned that way)
  3. The right circle is complete or also positioned similarly
  4. They are side by side horizontally
  5. The circles have white interiors (or transparent, but appear white against the background)

To recreate this in SVG:

  • I need two <circle> elements (or <path> or <ellipse>)
  • Thick stroke (probably around 10-15% of radius)
  • No fill (or white fill)
  • Black stroke
  • Positioned side by side

Looking at the proportions:

  • The circles seem to have roughly the same size
  • They are separated by some distance
  • The left one seems to be partially visible (maybe only the right half or two-thirds)
  • The right one seems fully visible or mostly visible

Actually, looking again, it seems the left circle is positioned so we see mostly its right side, and the right circle is positioned so we see its left side? No, wait. Let me reconsider.

The image shows:

  • Left side: a thick black arc that looks like the right side of a circle (or a full circle positioned partially off-screen to the left)
  • Right side: a thick black circle/ring that's positioned more to the right

Wait, looking carefully at the original image description and the visual content:

  • It appears to be two circles
  • The left circle is positioned with its left side cut off by the image boundary, showing mostly a "C" shape or partial circle on the left
  • The right circle is a complete ring positioned to the right
  • Or both are partial?

Actually, looking at the image again, it seems like:

  • Left element: A thick circular arc, open on the left side (like a "C" facing left, or rather, the right half of a circle)
  • Right element: A full circle or nearly full circle to the right

But wait, the left shape looks like it could be a full circle with the left side cut off by the image edge, and the right shape is a full circle.

Let me create an SVG with:

  • Two circles with thick strokes
  • Left circle centered at around (30, 50) with radius 40, stroke-width 8
  • Right circle centered at around (80, 50) with radius 40, stroke-width 8
  • ViewBox 0 0 100 100 (or similar)

Actually, looking at the aspect ratio, the image is wider than it is tall? No, the provided image seems to have a certain aspect ratio. Let me assume the viewBox is something like 0 0 200 100.

Better approximation:

  • Left circle: cx="50" cy="50" r="40" stroke-width="8"
  • Right circle: cx="150" cy="50" r="40" stroke-width="8"
  • But the left one is cut off, so maybe cx="20" so only the right part is visible?
  • Or maybe both are positioned so they overlap slightly or are separated?

Looking at the image once more:

  • The left circle appears to have its left side cut off by the image boundary
  • The right circle appears complete
  • They are separate, not touching
  • The left one is lower? No, they seem aligned vertically or nearly so.

Actually, looking at the bottom alignment:

  • The left circle seems to extend lower than the right circle? No, they look roughly aligned.

Let me go with:

  • ViewBox: proprietary 0 Circle 1: cx="25%" cy="50%" r="20%" stroke-width="5%"
  • Circle 2: cx="75%" cy="50%" r="20%" stroke-width="5%"

But to make it look like the image where the left one is cut off:

  • Circle 1: cx="20" cy="50" r="35" (so left edge is at -15, cut off)
  • Circle 2: cx="80" cy="50" r="35" (fully visible)

Wait, looking at the image provided, the left circle appears to be positioned so we see the right side of it (like a "D" shape), and the right circle is a full "O" shape. Actually, no - the left one looks like it might be a full circle with the left side cut off by the frame.

Let me create an SVG that captures the essence:

  • Two circles with thick black strokes
  • White fill (or transparent)
  • Left one partially visible (cut off on the left)
  • Right one fully visible
  • Vertical alignment roughly center
<svg width="200" height="100" viewBox="0 0 200 100" xmlns="http://www.w3.org/2000/svg">
  <!-- Left circle, partially cut off -->
  <circle cx="40" cy="50" r="35" fill="none" stroke="black" stroke-width="8" />
  <!-- Right circle -->
  <circle cx="140" cy="50" r="35" fill="none" stroke="black" stroke-width="8" />
</svg>

But looking at the image, the left circle might be positioned at around x=30 (if width is 100), and the right at x=70.

Actually, the image looks like it might be showing:

  • Left: A circle where only the right half is visible (cx=0 or negative)
  • Right: A full circle positioned to the right

Let me adjust:

  • Circle 1: cx="35" cy="50" r="40" (so left edge is at -5, cut off)
  • Circle 2: cx="120" cy="50" r="40" (fully visible)

Stroke width should be roughly 1/5 to 1/4 of the radius? Looking at the image, the stroke is quite thick. If radius is 40, stroke might be 8-10.

Let me provide a responsive SVG that approximates the visual:

<svg viewBox="0 0 160 100" xmlns="http://www.w3.org/2000/svg">
  <!-- Left circle (partially visible) -->
  <circle cx="35" cy="50" r="30" fill="none" stroke="black" stroke-width="8" />
  <!-- Right circle -->
  <circle cx="115" cy="50" r="30" fill="none" stroke="black" stroke-width="8" />
</svg>

This should create two thick-outlined circles, with the left one partially cut off on the left side if the viewBox starts at 0 and the circle extends to x=5 (35-30).

Actually, looking at the image one more time, I think the left circle is positioned such that its left side is cut off, showing a "C" shape (or rather, the right side of a circle), and the right circle is fully visible. The vertical position seems roughly centered.

Let me refine:

  • Left circle: cx=30, cy=50, r=25, stroke=6 (so left edge at 5, mostly visible but left side cut off)
  • Right circle: cx=80, cy=50, r=25, stroke=6

Wait, the spacing between them looks like roughly one radius or so.

Final check:

  • The image shows two black ring shapes
  • Left one appears to be cut off on the left edge of the image
  • Right one is fully visible
  • Both have same line thickness and diameter
  • They are side by side horizontally

So my SVG should be:

<svg width="300" height="150" viewBox="0 0 300 150" xmlns="http://www.w3.org/2000/svg">
  <circle cx="75" cy="75" r="60" fill="none" stroke="black" stroke-width="12" />
  <circle cx="225" cy="75" r="60" fill="none" stroke="black" stroke-width="12" />
</svg>

But with the left one positioned so its left side is cut off. So maybe cx="60" so left edge is at 0.

Actually, looking at the original image, the left circle seems to be positioned with roughly 1/4 to 1/3 of it visible? No, it looks like more than half is visible. Maybe 2/3 or 3/4 of the circle is visible on the left.

Let me go with:

  • Left circle: cx="50", r="40", so left edge at 10 (mostly visible)
  • Actually, looking at the crop, the left circle seems to extend from near the left edge to about the middle, and the right circle is from middle to right edge.

Wait, looking at the provided image crops:

  • First crop shows the left circle's right side and the gap between them
  • Second crop shows the right circle

So the left circle is indeed partially cut off on the left side of the image.

I'll provide an SVG with viewBox="0 0 200alpXD circle at (50,50) with r=40 and another at (150,50) with r=40, stroke-width around 8.

Actually, to make the left one cut off: cx=40, r=40 (so x from 0 to 80).

Final SVG:

<svg viewBox="0 0 200 100" xmlns="http://www.w3.org/2000/svg">
  <circle cx="···" cy="50" r="40" fill="none" stroke="black" stroke-width="8" />
  <circle cx="160" cy="50" r="40" fill="none" stroke="black" stroke-width="8" />
</svg>

With cx="40" for the left one (so it extends from 0 to 80), and cx="160" for the right one (80 to 160? No, 120 to 200). Wait, if cx=160 and r=40, it goes from 120 to 200.

Spacing: from 80 (right edge of left) to 120 (left edge of right) is 40 units gap. That seems like a big gap. In the image, the gap looks smaller than the radius.

Maybe:

  • Left: cx="45", r="35" (range: 10 to 80)
  • Right: cx="135", r="35" (range: 100 to 170)
  • Gap: 20 units
  • Stroke: 7

This looks better.

Or simply:

<svg viewBox="0 0 180 100" xmlns="http://www.w3.org/2000/svg">
  <circle cx···" cy="50" r="35" fill="none" stroke="black" stroke-width="8"/>
  <circle cx="130" cy="50" r="35" fill="none" stroke="black" stroke-width="8"/>
</svg>

Left circle at 45: range 10-80
Right circle at 130: range 95-165
Gap: 15 units (small gap)

Actually, looking at the image, the gap seems to be about one stroke-width or so, or maybe 1/4 of the diameter.

I'll provide a clean, simple approximation that captures the essence.

<svg viewBox="0 0 200 operativeCircle cx="45" cy="50" r="35" fill="none" stroke="black" stroke-width="8"/>
  <circle cx="140" cy="50" r="35" fill="none" stroke="black" stroke-width="8"/>
</svg>

@tempgidam
Copy link

tempgidam commented Feb 3, 2026

Okay, I replaced the first shard with the updated one (and removed the --chat-template-file parameter which was loading the modified template from the hf discussion thread to fix the thinking tags with the old version). Sadly, this doesn't seem to change anything for me. I'm getting the same results across the board.

I also tried some other things:
-redownloading/rebuilding the PR
-trying the other mmproj files (fp16, q8)
-the same setup I described above except with the UD-Q4_K-XL release by Unsloth
-updated ST for good measure
-tried switching the "Inline Image Quality" parameter in ST around from "High" to "Low" and back

However, nothing changed. The model still sees artifacts and seams across all of them.

@AesSedai
Copy link
Author

AesSedai commented Feb 3, 2026

Thanks both for the feedback, I'll continue to tinker with this and see if I can figure out the issue :)

@AesSedai
Copy link
Author

AesSedai commented Feb 4, 2026

@tempgidam / @z80maniac I've got something that might be worth testing on your setups.

In the following file: tools/mtmd/models/kimik25.cpp

diff --git a/tools/mtmd/models/kimik25.cpp b/tools/mtmd/models/kimik25.cpp
index d79b2f39c..6db47e2c9 100644
--- a/tools/mtmd/models/kimik25.cpp
+++ b/tools/mtmd/models/kimik25.cpp
@@ -26,7 +26,7 @@ ggml_tensor * clip_graph_kimik25::resize_position_embeddings_3d(uint32_t interpo
 
     pos_embd = ggml_permute(ctx0, pos_embd, 2, 1, 0, 3);
     pos_embd = ggml_interpolate(ctx0, pos_embd, height, width, n_embd, 1, mode);
-    pos_embd = ggml_permute(ctx0, pos_embd, 1, 2, 0, 3);
+    pos_embd = ggml_permute(ctx0, pos_embd, 2, 1, 0, 3);
     pos_embd = ggml_cont_2d(ctx0, pos_embd, n_embd, width * height);
     return pos_embd;
 }

Try swapping the numbers 1, 2, 0, 3 on line 29 to 2, 1, 0, 3, then recompile and test again please. I've tried both the RMS picture and the circle picture on my setup with this change and I'm not seeing any "double circle" or "glitch" thinking.

I might have goofed that permute and it's swapping w/h incorrectly before the ggml_cont_2d reshapes it.

@z80maniac
Copy link
Contributor

@AesSedai Yes, with this patch my test creates only one circle:

<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 200 200">
  <circle cx="100" cy="100" r="80" fill="none" stroke="black" stroke-width="16"/>
</svg>

And the viewBox is a square, not a rectangle, like in the previous test.

This SVG is almost exact representation of the original picture:

Details image

And in the reasoning it doesn't say that there are two circles:

The image is a simple black circle outline (a ring) on a white background. It's a circle with a thick stroke and no fill (or white fill).

@tempgidam
Copy link

@AesSedai
Yeah, I think this did the trick. I quickly went through my previous tests and the model no longer mentions anything odd about the image quality for any of them. The descriptions now match what I'm getting from the API very closely.

Thanks for your work!

@AesSedai
Copy link
Author

AesSedai commented Feb 4, 2026

Thanks for validating, I've pushed that change to the branch now. Much appreciated!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

examples python python script changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants